Getting started with Apama EPL

Introduction to Apama Event Processing Language

EPL is a flexible and powerful “curly-brace”, domain-specific language designed for writing programs that process events. In EPL, an event is a data object that contains a notification of something that has happened, such as a customer order was shipped, a shipment was delivered, a sensor state change occurred, a stock trade took place, or myriad other things. Each kind of event has a type name and one or more data elements (called fields) associated with it. External events are received by one or more adapters, which receive events from the event source and translate them from a source-specific format into Apama’s internal canonical format. Derived events can be created as needed by EPL programs.

Though it contains many of the familiar constructs and features found in general-purpose programming languages like Python or Java, EPL also has special features to make it easy to aggregate, filter, correlate, transform, act on, and create events in a concise manner. Here is the canonical “Hello world” example written in EPL:

monitor HelloWorld
{
   action onload()
   {
      print "Hello world!";
   }
}

The Apama event processor, called the correlator, receives events of various types from external sources. The EPL programs that process these events are monitors.

Monitors have registered event handlers, called listeners, for events of particular types with specific combinations of data values or ranges of values. When a listener detects an event of interest, it triggers a particular action. If there are no listeners for an event, the correlator either discards it or passes it to a listener specifically for events that have no handler. A monitor instance processes events on one correlator and can send events to communicate with other monitors on the same correlator or remote correlators.

Event handlers in EPL are conceptually similar to methods or functions used for handling user-interface events in other languages, such as Java Swing or SWT applications. In EPL, the correlator executes code only in response to events.

The correlator is capable of looking for hundreds of thousands of different events or different event patterns concurrently. When you write an EPL application, you write a set of monitors and then you inject or load them into a running correlator. As streams of events pass into a correlator, the monitors and their listeners watch for the events or patterns of events that you have specified as being of interest. There are a variety of actions that you can specify that you want the correlator to perform when a listener detects an event or event pattern of interest. For example, the most common action for a monitor is to generate and dispatch a message to an external receiver.

EPL is case-sensitive.

How EPL applications compare to applications in other languages

EPL is an event-oriented programming language, as opposed to an object-oriented language. Because EPL is part of an event-processing framework, it requires a different approach to decomposing the problem you want to solve.

EPL syntax is similar to other scripting languages. EPL has variables, data structures, conditions, and procedures (called actions in EPL). But EPL supports a paradigm that is different from that supported by other scripting languages:

• A monitor is the basic module in EPL programs.
• All communication is by means of message passing.
• All processing is triggered in response to events.
• Monitors spawn instances of themselves to generate multiple units of execution and/or to initiate parallel processing.

EPL requires a different way of developing applications.

About dynamic compilation in the correlator

EPL is dynamically compiled. You inject (load) EPL source files into a running correlator. The correlator compiles the files into optimized byte-code representations.

The EPL compiler is strict. There is no implicit type conversion. You cannot discard return values. To minimize the chance of runtime errors, your code must be explicit and not make assumptions. The correlator terminates execution of a program at the first runtime error.

The dynamic compilation approach removes the need for a byte code interpreter that supports older versions of byte code. Also, the correlator can apply new optimization techniques during byte code generation.

About the Apama development environment in Apama Plugin for Eclipse

Apama Plugin for Eclipse provides an integrated environment for developing Apama applications. The process of developing an Apama application is centered around an Apama project. In Apama Plugin for Eclipse, you create a project and then you use Apama Plugin for Eclipse to:

• Add and manage the component files that make up the application.
• Write the EPL for your application.
• Specify the connectivity plug-ins that are necessary for the application.
• Specify the configuration properties necessary for launching the application.
• Run and monitor the application.
• Export the initialization information necessary for deploying the application.
• Export your EPL files to a correlator deployment package (CDP).

See Using Apama with Apama Plugin for Eclipse for detailed information.

In addition to using Apama Plugin for Eclipse to create Apama projects, you can also do this using the apama_project command-line tool. See Creating and managing an Apama project from the command line for more information.

As you add components to your application, Apama Plugin for Eclipse automatically generates the boilerplate EPL code for the application’s standard features and launches the appropriate editor where you add the code to implement the component’s behavior.

A central Apama feature in Apama Plugin for Eclipse is the EPL editor. The EPL editor provides support for writing EPL, for example:

• Automatic EPL validation
• Content assistance
• Auto-completion
• Hovering over an event declaration displays the event’s type definition
• Automatic indenting and bracketing
• A separate panel shows the hierarchy of the EPL that appears in the editor
• Ability to define templates for frequently-used fragments of EPL

In Apama Plugin for Eclipse, you can examine the EPL files that are part of the Apama demo applications.

See Overview of developing Apama applications for more information.

Terminology

This topic provides a definition of each important EPL term. The definitions are organized into several groups.

Basic modules

Data types

Monitors

Events

Listeners

Streams

See also the above definitions for the stream data type, stream listener, and the from statement.

Defining event types

Conceptually, an event is an occurrence of a particular item of interest at a specific time. Examples of events include:

• A price of $100 for a share of IBM stock at noon on November 7, 2014
• Purchase of 1000 shares of IBM stock at $80 per share at 12:01 PM on December 12, 2014
• RFID tag 123-456-789 was scanned at 10:05 AM at loading dock 3
• Purchase order 55555 for 10,000 widgets sent to Acme Motor Supply
• TCP/IP address 123.4.56.789 just accessed server 5
• Container X was overfilled greater than 0.2 grams more than standard amount

An event usually corresponds to a message of some form. The correlator is designed to take in huge numbers of messages per second, and sift them for the events or patterns of events of interest. When the correlator detects interesting events or patterns it can undertake a variety of actions.

A correlator can receive events in several ways:

• You use Apama Plugin for Eclipse to send events from a file.
• From an adapter that receives an event from an external source. Apama adapters translate events from non-Apama format to Apama format.
• You run the Apama engine_send utility to manually send events into the correlator.
• A monitor generates an event within the correlator.
• You can write an application in C, C++, Java, or.NET that uses the Apama client API to send events into the correlator.

The correlator propagates information by sending events.

In EPL, each event is of a specific type. An event type has a name and a particular set of fields. Each field has a name and is one of a selection of types. Every event instance of a given event type has the same set and order of fields. For the correlator to process an event of a specific event type, it needs to have the event type definition for that type. Having the definition for an event type, lets the correlator

• Operate on the messages of that event type
• Create optimal indexing structures for finding events of that type that are of interest

An event type definition specifies the event type’s name and the name and type of each of its fields.

See also Specifying named constant values.

Allowable event field types

A field in an event can be any Apama type. See also Types.

Certain field types are valid only within a certain scope and you cannot pass events with such field types outside that scope. The details are as follows:

• context — When an event contains a context type field, you can send the event to other monitors within the same correlator but you cannot send the event outside the correlator. In other words, you can send or route the event. See Generating events.
• chunk, listener and stream — An event that contains one or more of these types of fields is valid only within the monitor that creates it. You cannot send, route, or enqueue an event that contains a field of type chunk, listener or stream.

If an event contains a chunk, listener, or stream field you cannot listen for that event.

For more information, see the description of event in the API reference for EPL (ApamaDoc).

Format for defining event types

In EPL, the format for an event type definition is as follows:

event event_type {
   [
   [ wildcard] field_type field_name; |
   constant field_type field_name := literal; |
   action_definition
   ]...
}

Syntax description

Example event type definition

For example, the EPL definition of an event type for simple financial stock price ticks might include the stock’s name and its price:

event StockTick {
   string name;
   float price;
}

To represent a specific instance of an event, use the following form:

event_type (field1_value, field2_value...)

For example, a StockTick event describing Acme’s new price of 55.20 looks like this:

StockTick("ACME", 55.20)

The reading order of fields in an event type definition and in instances of that event type must always match and is always left-to-right and then top-to-bottom. That is, "ACME" is the value of the name field and 55.20 is the value of the price field.

Working with events

After you define an event type, there are built-in methods you can call on it, and there are various ways that you can make that event available to monitors.

You can call a number of methods on any event type. For an overview of these methods, see the description of event in the API reference for EPL (ApamaDoc).

Making event type definitions available to monitors

A monitor must have information about the type definitions of the events that it processes. You can provide this information as follows:

• Define the event type in a separate file that contains only event definitions. An event type definition file has a .mon extension. It is still an EPL file even though it contains only event type declarations.

  You can define any number of event types in a single file. A common practice is to define the event interface to a service in a file that is separate from the implementation of that service. You might have a single event interface file and multiple implementations of services that process those event types.

• Define the event type in the monitor. Only instances of that monitor can process events of that type. Also, events of that type cannot be sent into the correlator from outside. When you define an event type inside a monitor it has a fully qualified name. For example:

  monitor Test
  {
     event Example{}
  }
  

  The fully qualified name for the Example event type is Test.Example and the toString() output for the event name is "Test.Example()".

• After the optional package specification, define the event type at the beginning of an EPL file that also defines monitors. All event type declarations must be before the monitor declarations. After you inject this file into the correlator, the following monitors can process events of that type:

  • All monitors that you define in the same file
  • All monitors that you inject after you inject the file that contains the event definition.

You might have a need for different event type definitions to have the same event type name. In this situation, define each event type in a different package. Then, in your monitor, use one of the following ways to make the appropriate event type definition available. In the monitor:

• Specify the fully qualified name of the event type, for example:

  com.apamax.test.Status

• After any package declaration and before any other declarations, specify a using declaration. For example:

  using com.apamax.test.Status;

  In your code, you can then simply refer to the Status event type.

Do not create EPL structures in the com.apama namespace. This namespace is reserved for future Apama features. If you inadvertently create an EPL structure in the com.apama namespace, the correlator might not flag it as an error in this release, but it might flag it as an error in a future release.

See also Name Precedence.

An event type definition must be injected into the correlator before a monitor that processes events of that type. After you inject an event type definition into the correlator, any monitor that you inject after that can process events of that type.

During development, when you use Apama Plugin for Eclipse to launch a project, it ensures that files are injected in the right order. When more than one project requires the same event definition file, do one of the following:

• In each project, declare an external dependency on the common event definition file. See Specifying dependencies for a single-user project.
• Create a project that contains the common event definition file. In each project that requires these event definitions, declare a dependency on the project that contains the common event definition file. See Specifying projects.

Channels and input events

Adapters, Apama client applications, and tools such as the engine_send correlator utility send events into the correlator. Each incoming event is associated with a channel either explicitly or implicitly. An event that has a channel explicitly set is delivered on the specified channel. An event that does not have a channel explicitly set is delivered on the default channel. The default channel’s name is the empty string.

An incoming event that is sent on the default channel goes to each public context. In addition, contexts can subscribe to channels of interest (see Subscribing to channels). An incoming event for which a channel is explicitly set goes to each context that is subscribed to its associated channel. If there are no contexts subscribed to the specified channel the event is discarded.

Events sent into the correlator from, for example, clients and adapters, are not normally delivered to external receivers. However, external receivers can specify the com.apama.input channel in their configuration. This is a wildcard for all events coming into the correlator. Also, an external receiver can specify com.apama.input.channel_name to receive correlator input events that are associated with that particular channel.

When two events are sent to different channels there is no ordering guarantee. The only guarantee is that events going from the same source to the same destination on the same channel will be delivered in order. Also, if there is an external connection with, for example, an adapter or client, then the events must use the same connection for them to be delivered in the same order.

All routable event types can be sent to channels, including event types defined in monitors.

An Apama application can use the Universal Messaging message bus to deliver events on specified channels. If a correlator is configured to connect to Universal Messaging, then a channel might have a corresponding Universal Messaging channel.

See Choosing when to use Universal Messaging channels and when to use Apama channels.